Mixed reality refers to the mixing of the real world, as depicted by video clip, with a virtual world to generate an environment in which real world and virtual world objects interact in real-time. For example, in a fitness gaming application, virtual avatars representing human runners may be embedded within a video, clip or live stream, that depicts an actual outdoor running environment, and the interaction of the virtual avatars through the real world environment provides a user with a realistic experience as though the user, displayed as the virtual avatar, is able to move and interact with the real-world environment in a realistic manner.